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Abstract 

We report the results of a search for flavor-changing neutral current, lepton-flavor 
violating, and lepton- number violating decays of D + , Df, and D° mesons (and 
their antiparticles) into modes containing muons and electrons. Using data from 
Fermilab charm hadroproduction experiment E791, we examine the nil and Kll 
decay modes of D + and Df and the l + l~ decay modes of D°. No evidence for any of 
these decays is found. Therefore, we present branching-fraction upper limits at 90% 
confidence level for the 24 decay modes examined. Eight of these modes have no 
previously reported limits, and fourteen are reported with significant improvements 
over previously published results. 
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The SU(2)xU(l) Standard Model of electroweak interactions qualitatively 
accounts for the known decays of heavy quarks and can often quantitatively 
predict the decay rates. However, this model is incomplete in that it does not 
account for the number of quark and lepton families observed, nor their hierar- 
chy of mass scales. Also unknown is the mechanism responsible for breaking the 
underlying gauge symmetry. One way to search for physics beyond the Stan- 
dard Model is to search for decays that are forbidden or else are predicted to 
occur at a negligible level. Observing such decays would constitute evidence 
for new physics, and measuring their branching fractions would provide in- 
sight into how to modify our theoretical understanding, e.g., by introducing 



new particles or new gauge couplings. 

In this letter we present the results of a search for 24 decay modes of the 
neutral and charged D mesons (which contain the heavy charm quark). These 
decay modesQ fall into three categories: 

(1) FCNC - flavor-changing neutral current decays (D° — > £ + £~ and Df d s -, — > 
h + £ + £~ , in which h is tt or K); 

(2) LFV - lepton-flavor violating decays (D° — > / u ± e T , Df ds \ — > h + fi ± e T , and 
DL| — > h~{i + e + , in which the leptons belong to different generations); 

(3) LNV - lepton-number violating decays {Df ds ^ —* h~£ + £ + , in which the 
leptons belong to the same generation but have the same sign charge). 

Decay modes belonging to (1) occur within the Standard Model via higher- 
order diagrams, but the estimated branching fractions are 10~ 8 to 10~ 6 [1]. 
Such small rates are below the sensitivity of current experiments. However, if 
additional particles such as supersymmetric squarks or charginos exist, they 
could contribute additional amplitudes that would make these modes observ- 
able. Decay modes belonging to (2) and (3) do not conserve lepton number 
and thus are forbidden within the Standard Model. However, lepton number 
conservation is not required by Lorentz invariance or gauge invariance, and a 
number of theoretical extensions to the Standard Model predict lepton-number 
violation [2]. Many experiments have searched for lepton-number violation in 
K decays, and for lepton-number violation and flavor-changing neutral cur- 
rents in D and B decays. The limits we present here for rare and forbidden 
dilepton decays of the D mesons are typically more stringent than those ob- 
tained from previous searches [3], or else are the first reported. 

The data are from Fermilab experiment E791 [4], which recorded 2 x 10 10 
events with a loose transverse energy trigger. These events were produced by 
a 500 GeV/c n~ beam interacting in a target consisting of five thin foils 
that had 15 mm center-to-center separation along the beamline. The most 
upstream foil was 0.5 mm thick platinum. It was followed by four foils consist- 
ing of 1.6 mm thick diamond. Momentum analysis was provided by two dipole 
magnets that bent particles in the horizontal (x-z) plane. Position information 
for track and vertex reconstruction was provided by 23 silicon microstrip de- 
tectors (6 upstream and 17 downstream of the target) along with 10 planes of 
proportional wire chambers (8 upstream and 2 downstream of the target), and 
35 drift chamber planes. The experiment also included electromagnetic and 
hadronic calorimeters, a muon detector, and two multi-cell Cerenkov coun- 
ters that provided n/K separation in the momentum range 6 — 60 GeV/c [5]. 
The kaon identification criteria varied by search decay mode. We typically 
required that the momentum- dependent light yield in the Cerenkov counters 



Charge-conjugate modes are included implicitly throughout this paper. 



be consistent with that of a kaon track measured in the spectrometer. 

Electrons were identified by an electromagnetic calorimeter [6] that consisted 
of lead sheets and liquid scintillator located 19 m downstream of the target. 
Electron identification was based on energy deposition and transverse shower 
shape in the calorimeter. The electron identification efficiency varied from 
62% for momenta below 9 GeV/c to 45% for momenta above 20 GeV/c. The 
decrease in efficiency with increasing momentum reflects the fact that higher 
momentum electrons populate a more congested region of the spectrometer. 
The pion misidentification rate was approximately 0.8%, independent of pion 
momentum. 

Muon identification was obtained from two planes of scintillation counters. The 
plane that measured vertical coordinates (y) consisted of 16 scintillation coun- 
ters, each 3 meters long and 14 cm wide. The plane that measured horizontal 
coordinates (x) consisted of 14 counters, each 3 meters long and covering a 
full width of 5.5 meters in the x-direction. The counters were located behind 
shielding with a thickness equivalent to 2.5 meters (15 interaction lengths) of 
iron. Candidate muon tracks projected into the muon system were required to 
pass a series of muon quality criteria that were optimized with D + —> K /i + ^ 
decays from our data [7]. Timing information from the y-coordinate counters 
was used to improve the position resolution in the x-direction. The efficiencies 
of the muon counters were measured in special runs using muons originat- 
ing from the primary beam dump, and were found to be (99 ±1)% for the 
y-coordinate counters and (69 ±3)% for the x-coordinate counters. The prob- 
ability for misidentifying a pion as a muon decreased as momentum increased, 
from about 6% at 8 GeV/c to (1.3±0.1)% for momenta greater than 20 GeV/c. 

After reconstruction, events with evidence of well-separated production (pri- 
mary) and decay (secondary) vertices were retained for further analysis. To 
separate charm candidates from background, we required the following: that 
secondary vertices be well-separated from the primary vertex and located well 
outside the target foils and other solid material; that the momentum vector 
of the candidate charm meson point back to the primary vertex; and that the 
decay track candidates pass approximately 10 times closer to the secondary 
vertex than to the primary vertex. A secondary vertex had to be separated 
from the primary vertex by greater than 20 a L for D + decays and greater 
than 12 o L for D° and Df decays, where a L is the calculated resolution of the 
measured longitudinal separation. In addition, the secondary vertex had to 
be separated from the closest material in the target foils by greater than 5 a' , 
where a' is the uncertainty in this separation. The sum of the vector momenta 
of the tracks from the secondary vertex was required to pass within 40 /im of 
the primary vertex in the plane perpendicular to the beam. Finally, the net 
momentum of the charm candidate transverse to the line connecting the pro- 
duction and decay vertices had to be less than 300 MeV/c for D° candidates, 



less than 250 MeV/c for D+ candidates, and less than 200 MeV/c for D + can- 
didates. These selection criteria and, where possible, the kaon identification 
requirements, were the same for the search mode and for its normalization 
signal. 

For this study we used a "blind" analysis technique. Before our selection crite- 
ria were finalized, all events having masses within a mass window AM5 around 
the mass of D + , D+, or D° were "masked" so that the presence or absence 
of any potential signal candidates would not bias our choice of selection cri- 
teria. All criteria were then chosen by studying signal events generated by a 
Monte Carlo simulation program (see below) and background events from real 
data. Events within the signal windows were unmasked only after this opti- 
mization. Background events were chosen from a mass window AM# above 
and below the signal window AM§. The criteria were chosen to maximize the 
ratio Ns/\^Nb, where N$ and Nb are the numbers of signal and background 
events, respectively. We used asymmetric windows for the decay modes con- 
taining electrons to allow for the bremsstrahlung low-energy tail. The signal 
windows are: 

1.84 < M(D+) < 1.90 GeV/c 2 for D + -> hfifi, 

1.78 < M(D+) < 1.90 GeV/c 2 for D+ -► hee and hfxe, 

1.95 < M(D+) < 1.99 GeV/c 2 for L>+ -> hfifi, 

1.91 < M(D+) < 1.99 GeV/c 2 for D+ -»• hee and fyue, 

1.83 < M(D°) < 1.90 GeV/c 2 for D° -> fj.fi, 

1.76 < M(D°) < 1.90 GeV/c 2 for D° -► ee and fxe. 



We normalize the sensitivity of our search to topologically similar Cabibbo- 
favored decays. For the D + decays we use D + — > K~7t + tt + ; for Df decays 
we use Df — > 07r + ; and for D° decays we use D° — > K~-k + . The widths of 
our normalization modes were 10.5 MeV/c 2 for D + , 9.5 MeV/c 2 for Df, and 
12 MeV/c 2 for D°. The events within the ~ 5 a window are shown in Figs. 
la-c. The upper limit for each branching fraction Bx is calculated using the 
following formula: 

R - Nx £Norm R ^9^ 

jVNorm ^X 

where Nx is the 90% CL upper limit on the number of decays for the rare or 
forbidden decay mode X, and ex is that mode's detection efficiency iV Norm is 
the fitted number of normalization mode decays; £Norm is the normalization 
mode detection efficiency; and B^ OTm is the normalization mode branching 
fraction obtained from the Particle Data Group [3]. 
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Fig. 1. Top row: typical charm signals in normalization modes used for the a) 
D + , b) Df, and c) D° decay modes. The signal region is shaded. Bottom row: 
invariant mass plots of D + candidate decays to d) i"f~/x + /i + , e) K~e + e + , and f) 
K~ /U + e + , showing reflections primarily from misidentified D + — > K~tt + tt + decays. 
These modes are not used to set upper limits but are instead used to estimate 
misidentification rates following the method described in the text. The solid curves 
are normalized Monte Carlo fits. The dashed lines show the signal window. 



The ratio of detection efficiencies is 



^Norm -''Norm 



£x 



N MC 



(3) 



where N$£ m and JV^ are the fractions of Monte Carlo events that are recon- 
structed and pass the final selection criteria, for the normalization and decay 
modes respectively. The simulations use Pythia/Jetset [8] as the physics 
generator and model the effects of resolution, geometry, magnetic fields, multi- 
ple scattering, interactions in the detector material, detector efficiencies, and 
the analysis selection criteria. The efficiencies for the normalization modes 
varied from approximately 0.5% to 2% depending on the mode, and the effi- 
ciencies for the search modes varied from approximately 0.1% to 2%. 



Monte Carlo studies show that the experiment's acceptances are nearly uni- 
form across the Dalitz plots, except that the dilepton identification efficiencies 
typically drop to near zero at the dilepton mass threshold. While the loss in ef- 
ficiency varies channel by channel, the efficiency typically reaches its full value 
at masses only a few hundred MeV/c 2 above the dilepton mass threshold. We 
use a constant weak-decay matrix element when calculating the overall de- 
tection efficiencies. Two exceptions to the use of the Monte Carlo simulations 
in determining relative efficiencies are made: those for Cerenkov identification 
when the number of kaons in the signal and normalization modes are different, 



and those for the muon identification. These efficiencies are determined from 
data. 
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Fig. 2. Final event samples for the D + (rows 1-3), Df (rows 4-7), and D (row 
8) decays. The solid curves represent estimated background; the dotted curves rep- 
resent signal shape for a number of events equal to the 90% CL upper limit. The 
dashed vertical lines are AMs boundaries. 

The 90% CL upper limits Nx are calculated using the method of Feldman and 
Cousins [9] to account for background, and then corrected for systematic errors 
by the method of Cousins and Highland [10]. In these methods, the numbers 



of signal events are determined by simple counting, not by a fit. All results 
are listed in Table 1 and shown in Figs. 2 and 3. The kinematic criteria and 
removal of reflections (see below) are different for the D + , Df, and D°. Thus, 
the D + and Df rows in Fig. 2 with the same decay particles are different, and 
the seventh row of Fig. 2 is different from the bottom row of Fig. 1. 

The upper limits are determined by both the number of candidate events and 
the expected number of background events within the signal region. Back- 
ground sources that are not removed by the selection criteria discussed earlier 
include decays in which hadrons (from real, fully-hadronic decay vertices) are 
misidentified as leptons. In the case where kaons are misidentified as lep- 
tons, candidates have effective masses which lie outside the signal windows. 
Most of these originate from Cabibbo-favored modes D + — > K~ir + ir + , D+ — > 
K~K + tt + , and D° — > K~n + (and charge conjugates). These Cabibbo-favored 
reflections are explicitly removed prior to the selection-criteria optimization. 
There remain two sources of background in our data: hadronic decays with pi- 
ons misidentified as leptons (-/VMism) and "combinatoric" background (iVc m b) 
arising primarily from false vertices and partially reconstructed charm decays. 
After selection criteria were applied and the signal windows opened, the num- 
ber of events within the window is iVobs = ^sig + ^MisiD + ^Cmb- 

The background NuisiD arises mainly from singly-Cabibbo-suppressed (SCS) 
modes. These misidentified leptons can come from hadronic showers reaching 
the muon counter, decays-in-flight, and random overlaps of tracks from oth- 
erwise separate decays ("accidental" sources). We do not attempt to establish 
a limit for D + — *■ K~£ + £ + modes, as they have relatively large feedthrough 
signals from copious Cabibbo-favored K~tt + tt + decays. Instead, we use the ob- 
served signals in K~£ + £ + channels to measure three dilepton misidentification 
rates under the assumption that the observed signals (shown in Figs, ld- 
f) arise entirely from lepton misidentification. The curve shapes were deter- 
mined from Monte Carlo simulations. The following misidentification rates 
were obtained: r m = (7.3 ± 2.0) x 10" 4 , r> = (2.9 ±1.3) x 1(T 4 , and 
r ee = (3.4± 1.4) x 10 -4 . Using these rates we estimate the numbers of misiden- 
tified candidates, N^f slD (for D + and D+) and N^ isID (for D°), in the signal 
windows as follows: 

Nffi m = r te -Nj® and < isID = r u ■ N^ s , (4) 

where iVggg and iVg^g are the numbers of SCS hadronic decay candidates 
within the signal windows. For modes in which two possible pion combinations 
can contribute, e.g., D + — ► /i + / u ± / u T , we use twice the above rate. These 
misidentification backgrounds were typically small or negligible. 

To estimate the combinatoric background iVcmb within a signal window AM$, 
we count events having masses within an adjacent background mass window 



AMb, and scale this number (N&m b ) by the relative sizes of these windows: 

AM S 
N C mb = ^- • N AMb . (5) 

To be conservative in calculating our 90% confidence level upper limits, we take 
combinatoric backgrounds to be zero when no events are located above the 
mass windows. In Table 1 we present the numbers of combinatoric background, 
misidentification background, and observed events for all 24 modes. 

The sources of systematic errors in this analysis include: statistical errors from 
the fit to the normalization sample A^orm; statistical errors on the numbers of 
Monte Carlo generated events for both N^ m and N^ ; uncertainties in the 
calculation of misidentification background; and uncertainties in the relative 
efficiency for each mode, including lepton and kaon tagging efficiencies. These 
tagging efficiency uncertainties include: 1) the muon counter efficiencies from 
both Monte Carlo simulation and hardware performance; 2) kaon Cerenkov 
identification efficiency due to differences in kinematics and modeling between 
data and Monte Carlo simulated events; and 3) the fraction of signal events 
(based on simulations) that would remain outside the signal window due to 
bremsstrahlung tails. The larger systematic errors for the Df modes, compared 
to the D + and D° modes, are due to the uncertainty in the branching fraction 
for the Df normalization mode. The sums, taken in quadrature, of these 
systematic errors are listed in Table 1. 



Table 1 
E791 90% confidence level (CL) branching fractions (BF) compared to PDG98 
limits. The background and candidate events correspond to the signal region only. 

(Est. BG) Cand. Syst. 90% CL E791 PDG98 [3] 

Mode iV C mb ^MisiD Obs. Err. Num. BF Limit BF Limit 
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In summary, we use a "blind" analysis of data from Fermilab experiment 
E791 to obtain upper limits on the dilepton branching fractions for flavor- 
changing neutral current, lepton-number violating, and lepton- family violating 
decays of D + , Df, and D° mesons. No evidence for any of these decays is 



10 



found. Therefore, we present upper limits on the branching fractions at the 
90% confidence level. These limits represent significant improvements over 
previously published results. Eight new D+ search modes are reported. A 
comparison of our 90% C.L. upper limits with previously published results [3] 
is shown in Fig. 3. 
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Fig. 3. Comparison of the 90% CL upper-limit branching fractions from E791 data 
(dark circles) with existing limits (open diamonds) from the 1998 PDG [3]. 
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